Conditional Expectation Function

Law of Iterated Expectations

Theorem

E_{X} [E_{Y} (Y ∣ X)] = E [Y]

Proof
离散型随机变量

\begin{aligned} E_{X} [E_{Y} (Y ∣ X)] & = \sum_{x} E_{Y} (Y ∣ X = x) P (X = x) \\ = \sum_{x} \sum_{y} y \cdot P (Y = y ∣ X = x) P (X = x) \\ = \sum_{x} \sum_{y} y \cdot P (X = x ∣ Y = y) P (Y = y) \\ = \sum_{x} P (X = x ∣ Y = y) \sum_{y} y \cdot P (Y = y) \\ = \sum_{y} y \cdot P (Y = y) \\ = E [Y] \end{aligned}

连续型随机变量

\begin{aligned} E_{X} [E_{Y} (Y ∣ X)] & = \int_{x} E_{Y} (Y ∣ X) p_{X} (x) d x \\ = \int_{x} \int_{y} y \cdot p_{Y ∣ X} (x, y) d y p_{X} (x) d x \\ = \int_{x} \int_{y} y \cdot p_{X ∣ Y} (x, y) p_{Y} (y) d y d x \\ = \int_{x} p_{X ∣ Y} (x, y) d x \int_{y} y \cdot p_{Y} (y) d y \\ = \int_{y} y \cdot p_{Y} (y) d y \\ = E [Y] \end{aligned}

Theorem
Let $g (X)$ be any function of $X$ , then

E [Y | X] = \underset{g}{\arg min} E [(Y - g (X))^{2}]

Proof

\begin{aligned} E [(Y - g (X))^{2}] & = E [(Y - E [Y | X] + E [Y | X] - g (X))^{2}] \\ = E [(Y - E [Y | X])^{2}] + E [(E [Y | X] - g (X))^{2}] + \\ + 2 E [(Y - E [Y | X]) (E [Y | X] - g (X))] \end{aligned}

the first term

E [(Y - E [Y | X])^{2}] = E {(Y - E [Y | X])^{2} | X} = E {V a r [Y | X]} \geq 0

the last term

\begin{aligned} 2 E [(Y - E [Y | X]) (E [Y | X] - g (X))] \\ = & 2 E [E {(Y - E [Y | X]) (E [Y | X] - g (X)) | X}] \\ = & 2 E [(E [Y | X] - g (X)) E {Y - E [Y | X] | X}] \\ = & 2 E [(E [Y | X] - g (X)) {E [Y | X] - E [Y | X]}] \\ = & 0 \end{aligned}

therefore, take

g (X) = E [Y | X]